Developing leading indicators of work injury and illness

Leading indicators of work injury and illness have the potential to help us identify and understand the factors affecting the risk of injury. They may also help identify ways to better prevent work injury and illness from occurring.

Workplaces use a variety of methods to prevent work injury and illness. These include: removing/reducing exposure to potentially harmful conditions; providing protective equipment; training supervisors and workers on identifying hazards and avoiding harm; and putting in place processes that enable workers and employers to work together to promote safety and health in the workplace.

How well safety efforts are working can be assessed by monitoring the rate of workplace incidents, the rate of serious injuries and the amount of work time lost as a result of injury. These are all examples of “lagging” or “trailing” indicators because they follow the programs, policies and practices that affect injuries and illnesses. Such data are important to monitor because evidence of increasing incidence of work injury and/or illness can be a signal that improvements are needed in the workplace safety system. However (and fortunately), many workplaces have too few injuries to be able to distinguish real trends from random occurrences. Also, it is possible that not all injuries are reported.

Many occupational health and safety (OHS) professionals are looking for “leading indicators”—organizational indicators that predict a higher risk of work injuries and illnesses before they occur, so that preventive steps can be taken to avert harm. Leading indicators may also be valuable to employers for benchmarking OHS practices to their industry peers, and to regulatory authorities for targeting resources to interventions likely to have the most impact.

This Issue Briefing explores the concept of leading OHS indicators, reviews some of the research in this area, and describes current research under way in Ontario, Canada, to develop a set of leading indicators of work injury and illness. Leading indicators are characteristics of workplaces (not of individual workers) that precede occupational health and safety outcomes and, if changed, are expected to change these outcomes. They are relevant to both the prevention of work injury and illness and the prevention and management of work disability in the event of injury.

Overview of leading indicator frameworks

The effort to identify leading indicators of work injury and illness has looked at several distinct (though related) influences, including safety culture, safety climate, the operation of joint labour-management health and safety committees, organizational policies and practices, and occupational health and safety management systems. Some measures reflect the intent to capture important elements represented in regulations, voluntary standards and workplace best practices.

There is a knowledge gap regarding what the most appropriate leading indicators are or should be. A scoping review of leading indicator measurement tools found little strong evidence for any leading indicator predicting injuries and illnesses (Brewer et al., 2008). The review noted that “researchers currently have no efficient way to identify organizational assessment tools or instruments for use in their occupational health studies.” In the following section, we review the most important ideas that have influenced thinking about the measurement of leading indicators.

Safety culture: Safety culture is thought of as a set of shared values and beliefs specific to safety that leads to observable behaviours, following Guldenmund (2000) and Hopkins (2006). Measuring safety culture—and correctly attributing observed safety behaviours to the shared values and beliefs in a workplace—has proven to be a major challenge (Schein, 1997). A review authored by Guldenmund (2000) found little consistent evidence to support the role of safety culture in preventing injuries, illnesses and work disability, noting that larger, more diverse samples are needed to better understand the relationship between culture and OHS outcomes.

Safety climate: Whereas safety culture refers to durable values and beliefs, safety climate refers to employee perceptions about safety at a point in time. Safety climate reflects a complex interplay of policies, procedures, norms and practices (Zohar, 2010). The U.S. National Institute for Occupational Safety and Health (NIOSH) has developed an eight-item measure of safety climate (Hahn and Murphy, 2008). Further research with this tool is required to clarify measurement properties and build an evidence base to support its practical application. Zohar champions an alternative approach to safety climate in which climate is assessed as the shared norms for acting safely while working. Supervisor, manager and worker perceptions are each assessed. A lot of studies use safety perception surveys. The evidence remains mixed on how well such measures of safety climate predict future occupational safety and health in an organization, and little welldesigned research has been carried out on whether a change in safety climate leads to a change in occupational safety and health outcomes. Because many practitioners consider safety climate to be an instantaneous assessment of safety culture, interest in this concept and the tools to measure it remains large. Some practitioners note that the Zohar tool, which was originally developed for the manufacturing sector, may be less relevant in other industrial sectors. The biggest challenge remains the need to reach a large number of members in an organization in order to assess its safety climate. Because climate reflects shared perceptions and not individual beliefs, safety climate researchers have rightly argued that, to validly assess safety climate, one must complete a comprehensive assessment of employees, supervisors and managers.

Joint labour-management health and safety committees: In some jurisdictions, health and safety legislation requires that joint health and safety committees (JHSCs) be in place in establishments over a certain size. (In Ontario, for example, the threshold is 20 employees.) These committees provide a forum for discussion of health and safety issues among representatives of workers and management. The evidence on the effectiveness of JHSCs in reducing injuries is limited. Geldart et al. (2010) found better health and safety outcomes among workplaces with stronger JHSC performance. Lewchuk, Robb and Walters (1996) found JHSCs had an effect on lost-time injuries. Clearly, JHSCs are a core component of any internal responsibility system, and should be a core component of any set of measures intended to capture leading indicators of organizational and management practices relevant to occupational health and safety. Nichol et al. (2009) have suggested one approach to assessing joint health and safety committee performance in health-care institutions.

Organizational policies and practices: Researchers have attempted to measure key workplace policies and practices that would be expected to affect OHS outcomes. The Organizational Policies and Practices Questionnaire (OPPQ) was developed based on the work of Hunt et al. (1993) and Habeck, Hunt and VanTol (1998) for use in high-risk manufacturing firms in Michigan. They showed that characteristics of the company environment, including management style, predicted workers’ compensation claims rates, as did elements of the safety program such as active safety leadership. The data collected by Habeck and Hunt were based on ratings by managers; a potential limitation of this approach is that it could tend to overstate the positives and understate the negatives for ratings of policies and practices. Amick et al. (2000), collaborating with Habeck and Hunt, modified the instrument, reducing it from 95 items to 52 items. They then administered it to a sample of 81 firms in Maine. An employee version of the questionnaire was also developed. The instrument involved seven scales: people-oriented culture; active safety leadership; safety training; safety diligence; ergonomics policies and practices; disability management; and labour-management climate. In a study of injured workers undergoing carpal tunnel surgery, the employee-answered OPPQ was the second strongest predictor of successful return to work and being absent from work (Amick et al., 2004). Ossmann et al. (2005) showed that key management informants and injured workers agreed on their responses to the questions. The 52-item instrument was then used in a sample of over 500 firms in the health-care, education and hotel sectors in Ontario by Williams (2005). Using the Maine and Ontario samples, the original seven scales and 52 items have been reduced to five scales and 26 items. The five scales are: safety practices; ergonomic policies and practices; disability management; active safety leadership; and people-oriented culture. While there is a growing literature on the OPPQ, no research has examined the predictive validity of these five scales since the early 1990s work of Hunt and Habeck.

Occupational health and safety management systems: Most OHS professionals would recognize occupational health and safety management systems (OHSMS) as a core element of the organization and management of worker health protection. OHSMS are distinct from the concepts of safety climate and safety culture. Fernandez-Muniz et al. (2007) proposed a new tool to measure OHSMS based primarily on a conceptualization of a safety management system in the process control industry. It includes: indicators of safety policy; worker incentives for participating in safety programs; safety training; communication; preventive planning; emergency planning; internal controls; and benchmarking techniques. The questionnaire was administered to a random sample of firms in Spain. Based on the results, the researchers reduced a 40-item scale to 29 items.

Challenges in developing leading indicators of OHS outcomes

Conceptual and technical challenges in the development of OHS leading indicators are common to all of the frameworks summarized in the previous section.

Conceptual clarity: Many of the individual questionnaire items that have been applied in the measurement of the different frameworks summarized in the previous section are similar. There are ongoing questions concerning the conceptual clarity of the different approaches to measuring leading indicators of occupational health and safety.

Exposures as leading indicators: A critical leading indicator would be the level of exposure to workplace hazards. Refined and rigourous measurements of hazard exposures are available from the disciplines of ergonomics and industrial hygiene. However, these measurement methods can be costly to undertake. A future challenge is to develop efficient methods of measuring hazard exposure that can be implemented easily across a representative sample of workplaces.

Measurement—external audits versus self-assessment: The process of measuring OHS leading indicators involves gathering information about workplace policies and practices. To obtain this information, one approach is to ask key workplace informants to assess policies and practices (self-assessment). An alternate approach is to rely on an external assessor or expert to gather information through some combination of document review, on-site observation and interviews, guided by an audit instrument. There remains little consensus on which approach is best. Many will consider external audits more valid than self-assessments. A recent review of research evidence on OHS management system audits by Robson and Bigelow (2010) found that research in this area is limited. The review noted that evidence on the inter-rater reliability of audits (whether different auditors have similar findings when auditing the same workplace) often found a low level of agreement among auditors. In the case of measurement based on self-assessment, there is a lack of consensus concerning the most appropriate informant within an organization. A common practice in OHS leading indicator research recruits managers when completing interviews or questionnaires (Amick, 2000).

Interpreting scores: Some of the instruments used involve ratings on a scale, such as 1 for “not at all” to 5 for “always” or “strongly.” How much better is a 5 than a 4? Consider, for example, an instrument like the Organizational Performance Metric outlined in the next section, which has eight such questions, so that total scores can range from 8 to 40. What is the relationship between different scores and OHS outcomes? Are there total score thresholds that demarcate different levels of performance? How much movement on the scale is necessary to capture a meaningful change?

Use of indicators to improve injury and illness prevention: Having reliable leading indicators of work injury risk may not necessarily improve prevention efforts. If a leading indicator shows a problem, does the organization respond to it? Conversely, if an organization goes through a change process to improve OHS, is this reflected in a change in the leading indicators?

Tailoring indicators to the context of a specific workplace: Having a common set of indicators used across workplaces would be helpful for jurisdictions interested in using a core set of indicators across sectors. However, there may be leading indicators of OHS performance that are specific to the industry or workplace context. What is the best way to balance these considerations?

Ontario Leading Indicators Project

Over the past several years, the Institute for Work & Health (IWH) has been working to develop leading indicators of OHS performance in partnership with four health and safety associations (HSAs) charged with providing training and consultation services to Ontario workplaces. The December 2010 report of the Expert Advisory Panel on Occupational Health and Safety noted this work and its potential value to the prevention system.

The initial project in Ontario, started in 2008, was to develop a short, easy-to-use measure. An eight-item questionnaire was developed by consultants from the HSAs. The questionnaire asks respondents to what extent the following statements are true:

1. Formal safety audits at regular intervals are a normal part of our business.
2. Everyone at this organization values ongoing safety improvement in this organization.
3. This organization considers safety at least as important as production and quality in the way work is done.
4. Workers and supervisors have the information they need to work safely.
5. Employees are always involved in decisions affecting their health and safety.
6. Those in charge of safety have the authority to make the changes they have identified as necessary.
7. Those who act safely receive positive recognition.
8. Everyone has the tools and/or equipment they need to complete their work safely.

The consultants considered these items, which were selected from a pool of questions, to be a reasonable set of questions to quickly assess an organization’s health and safety performance.

HSAs then administered the questionnaire to 642 workplaces. One respondent in each workplace was asked to assess the amount of time the eight practices occurred in his or her organization, from 0 per cent to 100 per cent (the five response categories being: 0-19; 20-39; 40-59; 60-79; 80-100).

Factor analyses and reliability tests showed the eight items are all required to measure organizational OHS performance (see www.iwh.on.ca/benchmarking-organizational-leading-indicators). Based on this assessment, a scale varying from 8 to 40 (eight questions each with five response categories from 1-5) was created, called the Organizational Performance Metric (OPM). Respondents’ answers were then matched to the workers’ compensation rates of their respective organizations over the previous 3.75 years. The results showed that, on average, the better the OPM score, the lower the injury and illness experience of the firm.

This scale is now being used in a larger survey of firms called the Ontario Leading Indicators Project (OLIP). The OLIP survey contains 17 measures within five tools: the OPM, the NIOSH safety climate tool; the Organizational Policies and Practices tool; the Occupational Health and Safety Management tool; and the Joint Health and Safety Committee Index (for more information on these tools, see www.iwh.on.ca/olip-survey-and-benchmarking-reports).

With the assistance of the HSAs, the survey has been completed by over 1,600 randomly selected firms registered with the Ontario Workplace Safety and Insurance Board (WSIB). The survey findings have been linked (with firm identity protected) to five years of WSIB claim history. This research is trying to identify:

• a set of scientifically supported metrics—leading indicators—relevant to all firms; and
• a model process for collecting unbiased benchmarking data (comparing individual workplace results to average results for organizations in the same sector).

Regarding the first objective, the research will examine the following for each measurement scale that is tested: internal consistency; test-retest reliability; the relationship of the measures to past workers’ compensation claim rates; whether firms that participate in the survey differ from those that choose not to; and who the best informant is in the firm to answer questions about OHS practices. IWH anticipates that findings from the survey will be available in the winter of 2013.

Conclusion

Leading indicators of occupational injury or illness could help workplaces identify factors affecting their risk of injury and take preventive steps to reduce this risk, as well as help them benchmark their OHS performance against the average for other, similarly situated firms. While companies talk about the importance of management and organizational initiatives, such as the presence of a strong safety culture or occupational health and safety management systems to promote good occupational health and safety, little scientific evidence is available to indicate which leading indicators, among many, should be used. To fill this gap, new research in Ontario seeks to identify the best and most usable set of management and organizational measures predictive of work injuries and illnesses.

References

1. Amick III BC, Habeck RV, Hunt A, Fossel AH, Chapin A, Keller RB, Katz JN. Measuring the impact of organizational behaviors on work disability prevention and management. Journal of Occupational Rehabilitation, 2000; 10(1):21-38.
2. Amick III BC, Habeck RV, Ossmann J, Fossel AH, Keller R, Katz JN. Predictors of successful work role functioning after carpal tunnel release surgery. Journal of Occupational Environmental Medicine, 2004; 46(5):490-500.
3. Brewer SB, King E, Delclos G, Irvin E, Lee L, Lewis C, Mahood Q, Spear J, Tetrick L, Williams R, Amick B. Workplace injury/illness and loss control program measurement tools. Toronto: Institute for Work & Health, 2008.
4. Fernandez-Muniz B, Montes-Peon JM, Vazquez-Ordas CJ. Safety management system: Development and validation of a multidimensional scale. Journal of Loss Prevention in the Process Industries, 2007; 20:52-68.
5. Geldart S, Smith CA, Shannon HS, Lohfeld L. Organizational practices and workplace health and safety: A cross-sectional study in manufacturing companies. Safety Science, 2010; 48:562-569.
6. Guldenmund FW. The nature of safety culture: A review of theory and research. Safety Science, 2000; 34:215-257.
7. Habeck RV, Hunt HA, VanTol B. Workplace factors associated with preventing and managing work disability. Rehabilitation Counseling Bulletin, 1998; 42(2):98-143.
8. Hahn, SE, Murphy LR. A short scale for measuring safety climate. Safety Science, 2008; 46:1047-1066.
9. Hopkins A. Studying organizational cultures and their effects on safety, Safety Science, 2006; 44:875-889.
10. Hunt HA, Habeck RV, VanTol B, Scully SM. Disability prevention among Michigan employers. Final report submitted to the Michigan Department of Labor. Upjohn Institute Technical Report No. 93-004. Kalamazoo, MI: W.E. Upjohn Institute for Employment Research, 1993.
11. Lewchuk W, Robb A, Walters V. The effectiveness of Bill 70 and joint health and safety committees in reducing injuries in the workplace: The case of Ontario. Canadian Public Policy, 1996; 22:225-243.
12. Nichol K, Kudla I, Manno M, McCaskell L, Sikorski J, Holness DL. Form and function of joint health and safety committees in Ontario acute care hospitals. Healthcare Quarterly, 2009; 12(2):86-93.
13. Ossmann J, Amick III BC, Habeck RV, Hunt A, Ramamurthhy G, Soucie V, Katz JN. Management and employee agreement on reports of organizational policies and practices important in return to work following carpal tunnel surgery. Journal of Occupational Rehabilitation, 2005; 15(1):17-26.
14. Robson LS, Bigelow PL. Measurement properties of occupational health and safety management audits: A systematic literature search and traditional literature synthesis. Canadian Journal of Public Health, 2010; 101 (Suppl 1):S34-S40.
15. Schein EH. Organizational culture and leadership, 2nd ed. San Francisco: Jossey-Bass, 1997.
16. Williams RM, Westmorland MG, Shannon H, Rasheed F, Amick III BC. Disability management practices in education, hotel/motel and health care workplaces. American Journal of Industrial Medicine, 2005; 47:217-226.
17. Zohar D. Thirty years of safety climate research: Reflections and future directions. Accident Analysis & Prevention, 2010; 42(5):1517-1522.